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Abstract 


This  research  effort,  sponsored  by  the  Program  Executive  Office  for  Air  ASW,  Assault,  and  Special 
Mission  Programs  (PEO(A)),  is  known  as  the  Navy  PEO(A)  Technical  Performance  Measurement 
(TPM)  System.  A  retrospective  analysis  was  conducted  on  the  T45TS  Cockpit-21  program  and  real-time 
test  implementations  are  being  conducted  on  the  Federal  Aviation  Administration’s  (FAA)  Wide  Area 
Augmentation  System  (WAAS)  program,  the  Navy’s  H-1  helicopter  upgrade  program,  and  is  currently 
under  consideration  for  other  test  implementations  across  the  Department  of  Defense  (DoD)  and  in 
private  industry. 

Currently-reported  earned  value  data  contains  invaluable  planning  and  budget  information  with  proven 
techniques  for  program  management,  however,  shortcomings  of  the  system  are  its  emphasis  on 
retrospection  and  lack  of  integration  with  technical  achievement.  The  TPM  approach,  using  the 
techniques  of  risk  analysis  and  probability,  offers  a  promising  method  to  incorporate  technical  assessments 
resulting  systematically  from  technical  parameter  measurements  to  derive  more  discrete  management 
data  sufficiently  early  to  allow  for  cost  avoidance.  Results  obtained  from  TPM  pilot  programs,  particularly 
the  Cockpit-21  program,  support  this  premise. 

Several  preliminary  issues  of  interest  and  conclusions  are  delineated  in  this  paper  that  demonstrate 
that  the  TPM  methodology  is  a  powerful  integrated  diagnostic  tool  in  support  of  the  new  paradigm 
advocating  a  multidisciplinary  approach  to  program  management.  It  also  promises  to  provide  a  powerful 
new  tool  in  proactive  risk  management. 


Introduction. 


In  recent  years  the  Department  of  Defense 
(DoD)  and  all  segments  of  the  American  economy 
have  been  under  increasing  pressure  to  change  the 
way  in  which  business  is  conducted.  This  condition 
is  the  result  of  a  number  of  converging  trends  and 
discrete  events— the  end  of  the  Cold  War,  a  political 
environment  skeptical  of  defense  expenditures  and 
active  international  involvement,  a  reduced  industrial 
base,  growing  international  competition,  evolving 
quality-focused  management  methodologies,  a  rapidly 
expanding  and  innovating  Information  Technology 
(IT)  community,  pressures  on  governments  to  reduce 
operations  and  balance  budgets— that  have  created  an 
environment  of  constant,  rapid,  and  unpredictable 
change  requiring  new  management  approaches  and 
techniques. 

For  the  government  systems  program  managers 
and  their  teams,  this  environment  creates  pressures 
that  are  translated  into  the  need  to  deliver  products 


using  best  value  analysis  with  cost  as  the  overriding 
determinant.  As  a  result,  information  is  needed  to 
allow  the  manager  to  make  informed  trade-off 
decisions  as  early  in  program  execution  as  possible. 

...the  cost  avoidance  window  of  opportunity 
is  before  the  fifteen  percent  mark  in  contract 
completion. 

In  addition,  any  condition  threatening  the  health 
of  program  development  must  be  identified 
sufficiently  early  to  allow  managers  to  mitigate  those 
areas  of  technical,  cost,  and  schedule  risk.  The  oft- 
repeated  rule-of-thumb  within  the  DoD  and  private 
industry  is  that  the  cost  avoidance  window  of 
opportunity  is  before  the  fifteen  percent  mark  in 
contract  completion.'  After  this  point,  the 
opportunity  for  the  avoidance  of  additional  resource 
consumption  is  greatly  diminished  and  mitigation 
focuses  upon  the  recovery  from  lost  effort  and 
remaining  cost  and  schedule  risk.  This  and  other 
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research  argues  for  concurrent  risk  identification  and 
reduction  beginning  in  the  concept  phase  of 
programs  and  carrying  through  engineering  and 
manufacturing  development? 

Current  acquisition  reform  initiatives  are  rapidly 
moving  program  management  teams  to  adoption  of  a 
holistic  approach  to  complex  systems  acquisition. 
Implementation  of  Integrated  Product  and  Process 
Development  (BPPD)  technique,  with  its  focus  on 
integration  of  program  management  activities  through 
multidisciplinary  Integrated  Product  Teams  (IPTs), 
has  established  the  cultural  and  structural  framework 
for  systems  thinking.  However,  few  tools  exist  to 
support  this  new  paradigm. 

While  traditional  cost  and  schedule  analysis, 
systems  engineering,  and  risk  management  provide 
the  program  management  team  with  a  broad  range  of 
tools,  many  of  these  techniques  are  derived  from 
separate  systems  that  are  viewed  in  isolation  from  one 
another.  Like  viewing  television,  each  separate  image 
flashed  before  the  program  manager  takes  on  an 
importance  and  reality  of  its  own,  providing  little 
context  relative  to  other  factors.  In  addition,  the 
signals  from  each  of  these  disciplines  are  being 
broadcast  over  separate  channels,  and  often  deliver 
contradictory  messages. 

The  perspective  of  many  of  these  traditional 
management  control  systems  is  also  retrospective  in 
nature,  documenting  history  rather  than  providing  the 
program  team  with  the  essential  information  needed 
for  day-to-day  management.  In  many  Earned  Value 
Management  Systems  (EVMS),  information  is 
normally  thirty  to  sixty  days  old  and  identify  cost  and 
schedule  variances  well  beyond  the  window  of 
opportunity  for  cost  and  schedule  risk  avoidance. 

These  systems  measure  work  accomplishment 
as  opposed  to  technical  achievement.^  As  a  parent  I 
provide  my  son  with  a  separate  two  week  allowance 
for  his  school  expenses.  Under  systems  that  measure 
work  accomplishment  based  on  time-phased 
budgeting,  his  successful  taking  of  an  exam  and  the 
expenditure  of  all  of  his  money  within  the  allotted  time 
would  earn  100%  value  regardless  of  the  grade  he 
achieved.  The  underlying  weakness  in  this  approach 
is  apparent. 

...  the  basic  tenets  of  the  process  are  the  need 
for  “seamless  management  tools”  that 
support  an  integrated  approach  ...  and 
“proactive  identification  and  management  of 
risk”  for  critical  cost,  schedule,  and  technical 
parameters  ...  (former  Secretary  Perry’s  memo  of 
May  1995) _ 


As  it  relates  to  program  management,  IPPD 
guidance  implicitly  acknowledges  these  deficiencies. 
In  former  Secretary  of  Defense  Perry’s  memo  of  May 
1995,  in  which  he  directs  the  use  of  IPPD  throughout 
the  DoD,  two  of  the  basic  tenets  of  the  process  are 
the  need  for  “seamless  management  tools”  that 
support  an  integrated  approach  to  program 
management  with  the  goal  of  enhancing  team 
decision-making,  and  “proactive  identification  and 
management  of  risk”  for  critical  cost,  schedule,  and 
technical  parameters  compared  against  best-in-class 
industry  benchmarks  that  provide  verification  of 
actual  achievement  of  technical  and  business-based 
parameters."^ 

In  support  of  IPPD,  and  as  of  this  writing,  DoD 
5000. 2-R  is  being  updated  to  require  Integrated 
Baseline  Reviews  (IBRs)  within  six  months  after 
contract  award  to  ensure  reliability  in  planning  and 
performance  measurement.^  Eor  those  program 
offices  that  have  conducted  an  IBR,  important  insight 
has  been  gained  by  members  of  the  IPT  into  the 
interrelationships  between  various  management 
control  systems  and  processes. 

Both  the  commercial  systems  engineering  and 
cost/schedule  analysis  communities  are  undergoing 
similar  change.  The  proposed  revision  to  EIA/IS-632, 
an  industry  systems  engineering  standard,  recently 
reviewed  at  the  annual  meeting  of  the  International 
Council  on  Systems  Engineering  (INCOSE),  includes 
Technical  Performance  Measurement  (TPM)  as  a 
critical  product  metric.®  In  addition,  within  the 
industry  standard  EVMS,  technical  performance  goals 
are  listed  as  necessary  indicators  to  be  used  in  order 
to  measure  programmatic  progress  among  its  32 
criteria.’ 

The  tools  required  to  support  the  demands  of 
this  new  environment  must  be  those  that: 

(a)  provide  an  integrated  view  across  programmatic 
elements; 

(b)  support  the  process  of  distributed  empowerment 
implicit  in  the  IPT  approach; 

(c)  logically  organize  data  resulting  from  systems 
engineering,  risk  management,  and  earned  value 
processes; 

(d)  provide  a  “real  time”  indication  of  contract 
performance  and  future  cost  and  schedule  risk; 

(e)  support  the  development  of  systems  thinking 
within  an  integrated  program  model. 

The  Program  Executive  Office  for  Air  Anti- 
Submarine  Warfare,  Assault  and  Special  Mission 
Programs  PEO(A)  TPM  system  is  a  promising 
methodology  that  addresses  these  goals  and  fits  well 
within  the  basic  tenets  of  IPPD  and  acquisition  reform 
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by  integrating  technical  performance  with  earned 
value  based  upon  programmatic  risk  assessments  and 
probability.  This  methodology  provides  a  flexible 
framework,  based  on  effective  business  practices,  that 
provides  government-contractor  program 
management  teams  with  the  information  they  need  to 
make  informed  management  decisions  at  critical 
milestones. 


Background. 


The  TPM  project  was  undertaken  in  1991  by  the 
PEO(A)  within  the  Naval  Aviation  Systems  Team 
(NAST).  From  its  earliest  inception,  the  PEO 
recognized  the  need  for  an  integrated  approach  to 
monitor  program  performance  based  upon  the  simple 
principle  that  the  solution  be  of  practical  utility  to  the 
program  office.  Consequently,  in  early  1991,  a  team 
consisting  of  representatives  from  each  PEO(A) 
program  office  was  organized  to  identify  and  validate 
a  process  for  the  integration  of  cost,  schedule,  and 
technical  performance  metrics.  After  several  meetings 
and  off-site  planning  conferences,  this  group 
generated  a  requirements  specification  that  became 
the  basis  for  the  project. 

This  document  identified  the  need  for  a 
standard  process  for  baseline  planning,  tracking,  and 
reporting  of  technical  performance  measurements  in  a 
manner  similar  to,  and  concurrent  with,  cost  and 
schedule  metrics.  In  addition,  the  document  specified 
the  need  for  a  means  of  determining  cost  and 
schedule  impacts  based  upon  technical  performance. 
In  1993,  the  TPM  Working  Group  selected  both  a 
proof-of-concept  and  commercial-off-the-shelf 
(COTS)  implementation  strategy  to  achieve  the  goals 
of  the  requirements  specification. 

The  proper  identification  of  technical 
performance  parameters  (TPPs)  and  the 
validity  of  technical  baseline  establishment 
were  seen  early  in  the  project  as  a  key  to  the 
proof  of  concept. 

The  proper  identification  of  technical 
performance  parameters  (TPPs)  and  the  validity  of 
technical  baseline  establishment  were  seen  early  in 
the  project  as  a  key  to  the  proof  of  concept.  The  Air 
Deployable  Active  Receiver  (ADAR),  a  sonobuoy 
program,  was  the  first  pilot  project  selected  to  test  the 
basic  premise  that  a  systematic  TPM  planning  and 
tracking  function  would  provide  an  early  warning, 
significantly  before  legacy  performance  measurement 
systems,  to  be  of  practical  benefit.  Insights  gained 
from  this  pilot: 


•  First,  cost  and  schedule  impact  assessments 
could  not  always  be  clearly  determined  because 
there  was  not  clear  linkage  between  technical 
parameters  and  budgeted  work  packages  via  the 
Work  Breakdown  Structure  (WBS). 

•  Second,  where  cost  and  schedule  impact 
assessments  could  be  made,  the  linkage  could  be 
made  at  a  fairly  high  level  within  the  WBS  (level 
4)  and  all  work  packages  could  be  associated 
directly  with  the  parameter. 

•  Third,  a  statistical  association  of  technical 
accomplishment  inserted  into  cost  and  schedule 
could  produce  calculated  impacts  amazingly 
close  to  what  was  eventually  experienced. 

•  Fourth,  the  identification  and  tracking  of 
technical  performance  metrics  in  a  disciplined  and 
systematic  fashion  provides  significant  early 
warning  of  potential  problems  and  their  nature. 

With  the  promising  results  from  the  ADAR 
program,  the  TPM  Project  Team  selected  the  Light 
Airborne  Multipurpose  System  (LAMPS)  Block  II 
Upgrade,  another  helicopter  program,  for  its  next  pilot. 
The  Block  II  program  was  selected  based  upon  its 
complexity,  its  high  dollar  value,  and  the  high 
technical  risk  inherent  in  the  effort.  The  TPM  team 
also  decided,  concurrent  with  this  pilot,  to  apply 
economic  utility  theory  as  the  means  for  determining 
the  technical  metric  that  would  be  used  for  calculating 
cost  and  schedule  impacts. 

The  results  from  LAMPS  Block  II  were  not  as 
encouraging  as  in  the  ADAR  pilot  but,  in  hindsight, 
of  greater  value: 

•  First,  the  technical  parameters  collected  were  of 
too  high  a  level  and  did  not  derive  from 
disaggregated  lower  level  parameters. 

•  Second,  the  practical  application  of  the  utility 
curve  assessment  approach  proved  both 
impractical  and  theoretically  unsound. 

•  Third,  the  overall  framework  of  estimating  cost 
and  schedule  impacts  using  the  “value”  of 
technical  progress  as  a  foundation  was  not 
flexible  enough  to  exploit  the  full  range  of  existing 
cost  estimating  techniques.  The  framework  relied 
solely  upon  expert  opinion— the  most  subjective 
method  in  the  cost  estimating  arsenal— eschewing 
through  the  approach  both  parametric  and 
industrial  engineering  measurement. 

While  pursuing  its  proof-of-concept  effort,  the 
TPM  Project  Team  formally  surveyed  both  the 
commercial  sector  and  other  government  program 
offices  to  determine  if  other  methodologies  or 
products  existed  that  would  meet  the  goals  of  the 
requirements  specification.  After  extensive  research. 
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only  one  untested  commercial  product  seemed  to 
show  promise.  This  product,  TCS  Integra  by 
Quantitech  Inc.  of  Huntsville,  Alabama,  was  selected 
for  test  implementation  and  the  results  from  its 
retrospective  pilot  implementation  on  a  PEO(A) 
program  are  presently  undergoing  review. 

By  the  fall  of  1995,  with  a  record  of  one  win  and 
one  loss,  and  an  apparent  wrong  turn  in  selecting 
economic  utility  theory  as  its  framework,  the  TPM 
Project  Team  conducted  a  thorough  reevaluation  of 
its  mission  and  methodology.  It  was  clear  that  if  the 
original  goal  of  the  requirement  specification  was  to 
be  met,  an  80%  solution  that  was  well-grounded  in 
each  of  the  disciplines  of  systems  engineering,  risk 
management,  and  cost/schedule  analysis  would  have 
to  be  found.* 


The  Key:  Systems  Thinking 
_ and  Probability. _ 


In  all  systems  where  optimum  performance  is 
necessary  or  desired,  it  is  useful  to  establish  what 
engineers  call  a  negative  feedback  system.  Negative 
feedback  is  the  basis  for  automatic  control  and 
regulation.  It  can  best  be  understood  by  a 
description  of  its  opposite,  which  is  positive 
feedback.  In  chemistry,  positive  feedback  usually 
takes  the  form  of  an  explosion.  In  program 
management,  positive  feedback  will  take  the  form  of  a 
program  requiring  greater  and  greater  commitment  of 
resources  for  achievement  of  requirements 
specifications  well  beyond  what  was  originally 
anticipated.* 

Most  U.S.  Navy  ships  still  use  steam  as  their 
main  source  of  propulsion.  In  the  1 8th  century  one  of 
the  roadblocks  to  the  effective  use  of  steam 
technology  was  the  inability  to  control  steam 
pressure.  This  inability  persisted  until  James  Watt 
(1736-1819)  and  the  Watt  steam  governor  came  along. 
The  principle  of  the  governor  was  to  create  an 
automatic  valve  that  would  regulate  the  flow  of  steam 
to  the  piston.  The  trick  was  to  link  the  valve  to  the 
rotary  motion  of  the  engine.  The  faster  the  engine 
moves,  the  more  the  valve  shuts  down.  The  slower 
the  engine  moves,  the  more  the  valve  opens  up.  The 
means  used  was  just  as  simple  and  elegant.  A  pair  of 
balls  on  hinged  arms  spin  around  using  the  principle 
of  centrifugal  force.  When  the  balls  spin  fast,  they 
rise  up  on  their  hinges;  when  spinning  slowly  they 
hang  down.  The  hinged  arms  are  linked  to  the  steam 
throttle. 


Our  feedback  systems  must  be  robust  enough 
to  give  us  a  discrete  indication  of  variability 
in  progress  to  allow  for  adjustments  to  be 
made. 

With  effective  tuning,  the  Watt  governor  keeps 
the  engine  turning  at  a  constant  rate  despite 
fluctuations  from  the  source  of  heat.  The  Watt  steam 
governor  was  responsible  for  the  effective  use  of 
steam  in  industrial  production,  giving  rise  to  the 
industrial  revolution,  and  to  the  creation  of  great 
navies  that  could  navigate  the  oceans  independent  of 
the  wind. 

What  the  program  manager  needs  is  the 
equivalent  of  a  Watt  steam  governor.  If  we  view  a 
program  as  a  system,  what  we  see  are  resources  as 
our  inputs  (in  terms  of  money,  time,  and  expertise) 
with  the  end  item  (e.g.,  a  ship,  aircraft,  or  satellite)  as 
our  output.  Our  feedback  systems  must  be  robust 
enough  to  give  us  a  discrete  indication  of  variability 
in  progress  to  allow  for  adjustments  to  be  made. 
Preferably,  our  feedback  systems  will  be  negative 
ones,  but,  as  we  all  should  know,  social  systems,  of 
which  a  program  is  one,  are  more  complicated  than 
our  analogy  to  a  relatively  simple  steam  plant. 

A  program,  as  an  organization,  is  a  type  of 
complex  adaptive  system.  A  complex  adaptive  system 
is  one  that  acquires  information  about  its  environment 
and  its  interactions  within  the  environment,  identifies 
information  of  importance,  places  that  information 
within  the  context  of  a  contextual  framework,  model, 
or  “schema,”  and  then  acts  on  the  basis  of  that 
schema.**  The  individual  members  of  the  program 
office— people— act  as  complex  adaptive  systems 
themselves  and  exert  a  powerful  influence  on  the 
selection  of  both  schema  and  those  adaptive 
pressures  that  are  used  in  making  decisions.  The 
extent  to  which  their  learning  brings  about  adaptive  or 
maladaptive  behavior  will  determine  the  survival  or 
failure  of  the  organization.** 

In  constructing  a  negative  feedback  system  for 
a  complex  adaptive  system,  an  understanding  of  the 
schema  and  context  in  which  the  system  functions  is 
necessary.  Also,  the  model  should  be  as  simple  as 
possible  and  only  contain  those  elements  absolutely 
necessary  for  approximating  reality. 

The  IPPD  technique  provides  the  necessary 
schema  around  which  to  construct  our  tools,  with  its 
emphasis  on: 

•  decentralized  authority  through  the  IPTs, 

•  the  renewed  importance  of  cost  as  the 
independent  variable. 
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•  the  use  of  performance  specifications  in 
acquisitions,  and 

•  the  emphasis  on  advance  planning  and  quality  as 
a  by-product  of  the  work  performed. 

That  these  business  practices  are  becoming 
universal  in  both  government  and  private  industry 
also  lends  us  valuable  insight. 

Stephen  Jay  Gould,  the  noted  Harvard 
polymath,  in  his  book  Full  House,  in  using  the 
disappearance  of  the  .400  hitter  from  baseball  as  his 
subject  to  demonstrate  increasing  excellence  of  play, 
illustrates  that  complex  systems  tend  to  organize 
themselves  as  a  set  of  probable  outcomes,  often 
within  a  normal  distribution.  Variation  in  this 
distribution  changes  over  time  as  members  become 
familiar  with  their  environment.  Gould  concludes  that 
(a)  complex  systems  improve  when  the  best 
performers  play  by  the  same  rules  for  extended 
periods  of  time,  and  (b)  as  play  improves  and  bell 
curves  march  toward  the  right  wall,  variation  must 
shrink.*^  Implicit  in  these  conclusions  is  the  effective 
ability  of  members  of  the  complex  system  to  learn  and 
adapt. 

Foundation  ...  With  the  adoption  of  earned 
value  management  and  critical  path  scheduling  as 
industry  standards,  the  foundation  was  laid  in  the  fall 
of  1995  for  the  TPM  Project  Team  to  use  the 
principles  of  systems  thinking  and  apply  them  to  the 
existing  disciplines  of  cost/schedule  control,  risk 
management,  and  systems  engineering  to  create  an 
integrated  diagnostic  tool.  Also,  the  rapid  advances 
in  desktop  computing  power,  even  within  the  short 
life  of  the  project,  brought  with  it  the  ability  to  cost- 
effectively  integrate  these  concepts. 

First  choice  ...  The  first  choice  was  to  ensure 
that  the  methodology  was  integral  to  existing 
processes  involved  in  planning  and  tracking  program 
performance,  and  supported  the  cultural  and 
structural  processes  established  under  IPPD.  It  was 
decided  that  both  a  TPM  process  and  a  practical, 
interactive  tool  would  be  developed  to  facilitate  an 
understanding  of  technical,  cost,  and  schedule  risk 
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issues. 

Second  choice  ...  The  second  choice  to  make 
was  to  select  the  way  in  which  technical  performance 
impacts  would  be  expressed.  The  team  decided  that, 
with  the  emphasis  on  cost,  the  industry  EVMS  would 
be  used  as  the  user  interface.  This  approach  had 
worked  well  on  ADAR.  This  meant  that  the  Budgeted 
Cost  for  Work  Performed  (BCWP),  or  “earned  value,’’ 
would  need  to  be  informed  by  technical  achievement, 
but  in  a  way  that  would  lend  credence  to  the 
projected  impact. 


Approaches  ...  Within  the  risk  management 
discipline,  there  are  basically  two  general  approaches 
for  estimating  cost  impacts— probabilistic  and 
deterministic.  A  probabilistic  approach  is  top-down, 
based  upon  the  probability  of  outcomes.  The  Monte 
Carlo  analysis  model  is  a  good  example  of  a 
probabilistic  approach.  A  deterministic  approach  is 
bottom-up,  based  upon  a  sequence  of  causes. 
Learning  curve  estimation  is  a  deterministic  method, 
though  it  still  possesses  a  large  element  of 
probability.  Probabilistic  models  are  by  nature 
inexpensive  to  apply  but  sometimes  lack  credibility. 
Deterministic  models  are  more  expensive  to  apply  but 
credibility  is  also  an  issue  if  the  work  is  not 
disaggregated  properly.  Also,  as  noted  above,  no 
model  is  ever  completely  deterministic— the  proper  mix 
between  the  two  approaches  must  be  selected. 

This  last  point  goes  to  the  heart  of  the  approach 
eventually  selected.  Risk  determination  is,  by  nature, 
probabilistic.  As  we  noted  above,  complex  systems 
tend  to  organize  themselves  in  a  normal  distribution 
of  likely  outcomes.  As  procedures  and  practices 
become  standard,  the  best  performers  tend  to  follow 
the  general  trend  toward  excellence  and  variation 
around  the  mean  shrinks.  Other  distributions  apply  in 
less  mature  environments,  but  a  statistical  tool  using 
the  assumption  outlined  above  as  its  basis  should 
meet  the  requirement  of  providing  sufficient  early 
warning  of  technical  perturbations  in  program 
development  as  long  as  the  technical  metrics  are 
derived  systematically,  a  planning  baseline  is 
established,  and  technical  performance  parameters  are 
disaggregated  and  properly  tied  to  the  WBS.'® 

Earned  Value  calculation  ...  With  the 
assistance  of  Naval  Reserve  Unit  NAVAIRSYS  1187 
under  the  command  of  CAPT  A.  R.  Pagnotta,  USNR-R 
(now  retired),  a  unit  consisting  of  information 
technology  and  systems  engineering  professionals, 
statisticians,  and  mathematicians,  the  TPM  Project 
Team  reengineered  the  TPM  method  of  calculating 
technical  earned  value.  Technical  earned  value  would 
be  the  key  metric  used  in  recalculating  BCWP  and 
also  used  in  the  algorithm  to  calculate  schedule 
impacts. 

Gantt  charts  are  the  standard  tracking  tool 
linking  program  activities  with  time.  Each 
TPP  normally  has  a  progress  plan  assigned 
to  it  based  on  how  the  development  activities 
will  be  performed. 

It  is  a  common  commercial  practice  to  segment 
work  into  key  product  development  paths,  or 
technical  progress  plans,  assignable  to  specialties 
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within  a  function  or  functions.  Gantt  charts  are  the 
standard  tracking  tool  linking  program  activities  with 
time.  Each  TPP  normally  has  a  progress  plan 
assigned  to  it  based  on  how  the  development 
activities  will  be  performed.'’ 

With  this  in  mind,  the  team  selected  as  its 
method  for  calculating  technical  earned  value: 

•  A  90-50-10  risk  profile,  that  is  equivalent  to  the 
probability  of  successfully  achieving  the  next 
TPM  milestone  [Pr(S)]. 

•  The  profile  is  then  applied  at  each  assessment 
date,  which  could  be  monthly,  or  some  other 
period.'* 

This  approach  isolates  technical  performance,  in 
terms  of  technical  achievement  and  deviation,  from 
cost  and  schedule. 

The  means  of  establishing  the  90-50-10 
probability  distribution  exploits  standard  risk 
management  estimation  techniques  based  on  analogy, 
parametrics,  and  industrial  estimation,  and  is 
constructed  concurrent  with,  and  integral  to,  the 
establishment  of  the  Systems  Engineering 
Management  Plan  (SEMP).  The  methodology  of 
using  technical  progress  plans  and  applying  a  risk 
profile  against  each  assessment  date  is  similar  to  the 
establishment  of  a  formal  baseline  for  WBS  work 
package  budgets  and  schedules.  The  baselining 
issues  raised  through  this  process  are  then  of  service 
during  the  IBR. 

Once  again,  the  application  of  systems  thinking 
is  instructive  in  understanding  the  application  of  the 
probability  distribution.  A  popular  analogy  concerns 
placing  a  monkey  in  front  of  a  typewriter.  According 
to  this  story,  given  enough  time,  the  monkey  would 
eventually  produce  the  collected  works  of 
Shakespeare.  Unfortunately  for  the  analogist, 
systems,  even  live  ones,  do  not  work  this  way.  Rare 
is  the  sudden  act  of  creation  from  whole  cloth. 
Richard  Dawkins,  the  Charles  Simonyi  Chair  of  Public 
Understanding  of  Science  at  Oxford,  limited  his 
simulated  computer  monkey  to  producing,  in  a  single 
random  step,  the  sentence  uttered  by  Polonius  in  the 
play  Hamlet:  “Methinks  it  is  like  a  weasel.”  The  odds 
of  getting  it  in  a  single  step  is  about  1  in  10,000  million 
million  million  million  million  million— requiring  a 
longer  time  to  achieve  than  all  of  the  time  that  has 
expired  since  the  beginning  of  the  universe.  When, 
however,  the  monkey  used  cumulative  progress,  built 
from  previous  steps  in  achievement,  the  computer 
built  the  target  phrase  in  generation  43  on  the  first  run 
and  in  generation  64  on  the  second  run.'® 

This  example  demonstrates  that  establishing  the 
probability  of  successfully  achieving  the  next 


technical  performance  milestone  is  the  proper 
approach  in  deriving  a  technical  earned  value.  The 
probability  of  successfully  achieving  an  end  goal  in  a 
single  step  is  vanishingly  small  and,  if  applied  in  a 
methodology,  will  give  us  an  overstated  negative 
impact.  In  our  approach,  however,  each  Pr(S) 
represents  a  discrete  event  along  our  progress  plan 
that,  when  combined  with  previous  scores,  gives  us 
an  assessment  of  the  cumulative  achievement  along 
the  development  path.  Breaking  down  the  path  into 
these  discrete  probability  assessments  also  has  the 
effect  of  isolating  and  reducing  subjectivity. 
Development  is,  after  all,  an  evolutionary  process 
built  on  the  cumulative  effort  expended  toward  the 
achievement  of  eventual  program  goals. 


An  Integrated  Diagnostic  Tool. 


Having  resolved  the  major  issues  of  its  bottom- 
up  review,  in  November  1995  the  TPM  Project  Team 
concurrently  pursued  the  development  of  both  a 
methodology  and  software  application  to  achieve  the 
integration  of  cost,  schedule,  and  technical 
performance.  Several  additional  meetings  resulted  in 
the  development  of  a  general  framework  that  would 
use  the  existing  internal  management  methodologies 
of  prime  contractors,  as  much  as  practicable,  and  to 
reorganize  existing  data  in  a  way  to  achieve  the 
desired  integration. 

The  final  result  of  these  meetings  was  a 
methodology  consisting  of  three  phases.  Eigure  1 
provides  an  overview  of  the  entire  methodology. 


Eigure  1 :  TPM  Methodology  Overview 
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1.  Select  Technical  Parameters.  Concurrent  with 
the  formulation  of  the  SEMP,  the  following  criteria  are 
used  to  assist  in  the  selection  of  critical  TPPs: 

•  Those  that  are  program  cost  drivers,  reside  on 
the  critical  path  schedule,  or  that  represent, 
based  on  formal  assessment,  high  risk  to  the 
program. 

•  Once  selected,  TPPs  are  organized  in  a 
weighted  hierarchy  to  establish  relative 
importance  and  interrelationships. 

•  Linkage  is  made  to  the  contract  WBS.  This  last 
activity  is  accomplished  in  order  to  obtain  a 
“technical  budget  baseline,”  or  the  budget 
associated  with  the  work  packages  that  are 
responsible  for  a  particular  parameter’s 
developmental  success,  and  that  would 
ultimately  be  placed  at  risk  should  performance 
not  meet  expectations. 

2.  Plan  TPM  Progress.  The  second  phase  is  to 
baseline  technical  performance  measurement  through 
the  establishment  of  a  technical  progress  plan  for 
each  TPP.  The  approach  to  planning  and  baselining 
TPM  is  virtually  the  same  as  that  used  in  baselining 
cost  and  schedule  measures  with  the  common  goal  of 
establishing  a  framework  from  which  to  assess  actual 
progress  and  measure  relative  performance.  A 
disaggregate  approach  is  used  to  reduce  subjectivity 
by  developing  progress  plans  for  lower  level  TPPs 
and  applying  the  cumulative  scores  from  these  lower 
level  plans  to  higher,  summary  level,  TPPs.  Figure  2 
shows  a  typical  progress  plan  for  the  weight  of  a  key 
component. 


Sample  Progress  Plan 
{weight  -  lbs  -  is  being  measured) 


Figure  2:  Sample  Progress  Plan 

In  the  Gantt  chart  used  as  our  example,  the 
technical  assessment  activity  dates  are  displayed 
along  the  horizontal  axis  and  the  units  of  measure 
are  on  the  vertical  axis.  The  straight  line  at  the  32.5 
pound  mark  represents  the  end  goal.  The  lower 


downward  sloping  line  marked  by  open  triangles  is 
the  technical  progress  plan  while  the  bolder  upper 
sloping  line  marked  by  filled  triangles  traces  the 
actual  achievement.  This  component  was  considered 
to  be  critical  to  the  end  item  design  and  a  bellwether 
of  overall  aircraft  weight. 

A  TPP  may  be  any  function,  physical 
characteristic,  design  goal,  or  other  aspect  of 
a  project  that  has  been  defined  by  the 
requirements  of  the  program. 

Looking  at  our  example  in  isolation  from  other  factors 
and  other  technical  parameters  points  out  the 
weakness  of  a  reductionist  approach  to  program 
management  in  which  TPMs  are  collected  apart  from 
other  activities.  The  importance  of  the  technical 
variance,  that  is,  the  space  between  the  open  and 
closed  triangles  is  unknown  and  can  easily  be 
dismissed.  Presently,  an  assessment  of  this  variance 
relies  too  heavily  on  expert  opinion.  Consequently, 
this  process  is  highly  subjective  and  leaves  open  the 
possibility  of  a  “rubber”  technical  baseline— one  in 
which  the  significance  of  any  variance  can  be 
misinterpreted.  This  is  the  importance  of  establishing 
a  TPP  hierarchy  and  weighting  activities  relative  to 
each  other  in  an  IPT  environment. 

The  weight  example  provided  is,  of  course, 
highly  simplified  and  given  only  as  an  analogue  for 
development  metrics  that  may  be  used  to  track 
technical  progress.  A  TPP  may  be  any  function, 
physical  characteristic,  design  goal,  or  other  aspect  of 
a  project  that  has  been  defined  by  the  requirements  of 
the  program.  Both  process  and  product  metrics  are 
candidates  that  may  be  selected  as  TPPs.  In  software 
development,  our  experience  indicates  that  qualitative 
process  metrics,  such  as  staffing  and  error  reporting, 
are  more  reliable  indicators  of  poor  technical 
achievement  than  more  traditional  product  metrics 
such  as  source  lines  of  code. 

3.  Determine  Risk.  The  third  and  last  phase  is  to 
apply  the  90-50-10  risk  profile  to  each  planned  value 
within  the  technical  progress  plan  to  serve  as  a 
benchmark  against  actual  achievement.  Insertion  into 
earned  value  is  accomplished  by  applying  the 
confidence  factor  as  the  technical  performance  score 
to  calculate  a  technically  informed  BCWP.  Critical 
path  schedule  impacts  are  calculated  similarly  by 
applying  the  confidence  factor  as  the  achievement 
metric  against  the  portion  of  the  schedule  placed  at 
risk  by  the  technical  parameter(s). 
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Figure  3:  Progress  Plan  with  Risk  Profile 


Figure  3  above  illustrates  the  technical 
variance  for  our  component  weight  example.  A 
probability  distribution  is  applied  to  every 
assessment  date  on  the  progress  plan,  which  has 
been  rotated  so  that  the  original  y-axis  is  horizontal. 
In  this  example,  the  second  assessment  date  is 
magnified  to  show  in  detail  the  relationship 
established  between  the  actual  measures  and  the 
assessed  probabilities  of  success  [Pr(S)].  The  actual 
measure  is  above  the  expected  value,  a  potentially 
unfavorable  condition  when  measuring  weight 
reduction.  In  this  case  the  actual  measure  falls  at 
the  90  percent  confidence  level.  Interpolation  is 
used  to  calculate  values  that  fall  on  intermediate 
points  along  the  slope  of  the  distribution. 


In  this  particular  example,  a  single-sided 
probability  distribution  is  applied  since  our  area  of 
interest  is  only  in  the  area  above  the  technical 
progress  plan.  In  cases  where  a  variance  significantly 
above  or  below  the  expected  value  is  an  indicator  of 
poor  technical  achievement,  such  as  in  the  case  of 
software  problem  reports  or  staffing,  a  double-sided 
distribution  is  applied  to  determine  the  earned 
technical  value. 

Distributions  may  be  customized  based  upon 
any  standard  analogous,  parametric,  or  industrial 
engineering  technique.  In  some  cases,  a  tolerance 
band  is  wider  at  the  beginning  of  the  technical 
activity  and  becomes  tighter  over  time.  Should 
customization  not  be  applicable  in  every  case,  a 
default  normal  distribution  is  applied  that  establishes 
the  90  percent  confidence  factor  to  actuals  at  5 
percent  deviation  from  expectation,  50  percent 
confidence  at  10  percent  deviation  from  expectation, 
and  10  percent  confidence  at  20  percent  deviation 
from  expectation. 

The  technical  earned  value  is  inserted  into 
EVMS  by  multiplying  the  confidence  level  by  the 


BCWS  for  the  WBS  elements  associated  with  the 
TPP.  For  example,  if  the  cumulative  BCWS  for  a  WBS 
element  is  $50,000,  and  the  cumulative  technical 
earned  value  is  50  percent,  then  the  calculated  BCWP 
is  $25,000.  Once  BCWP  is  calculated,  performance 
indices,  variances,  and  estimates  at  complete  can  be 
calculated.^® 

Critical  path  schedule  impacts  are  also 
calculated  by  using  the  confidence  level  metric.  This 
is  accomplished  by  taking  the  baseline  schedule  for 
the  associated  activity  and  multiplying  the  portion  of 
the  activity  placed  at  risk  by  the  confidence  level 
subtracted  from  100  percent  confidence.  For  example, 
a  scheduled  activity  is  90  days  duration.  If  the 
cumulative  confidence  level  is  90  percent  and  all  90 
days  of  the  activity  is  placed  at  risk  by  the  associated 
technical  performance  activity,  then  a  10  percent  lost 
effort  is  calculated  to  project  a  possible  slip  of  nine 
days  for  the  activity.  Using  standard  COTS  critical 
path  scheduling  tools,  impacts  for  associated 
activities  can  be  calculated  automatically. 

The  Cockpit-21  program  provided  an 
excellent  analysis  environment  with  several 
years  of  Cost  Performance  Reports  (CPRs) 
and  technical  performance  data  that  included 
comprehensive  planning  and  reported 
actuals. 


Proving  the  Methodology:  The  T45TS 
Cockpit  21  Retrospective  Analysis 


Once  settling  on  a  methodology,  questions 
arose  for  the  TPM  Project  Team  concerning 
interfacing  the  TPM  software  tool  with  EVMS 
software  packages.  Performance  Analyzer  (PA),  the 
standard  government  software  tool,  proved  not  to  be 
Windows  capable  and  did  not  offer  all  of  the  features 
desired  for  calculating  earned  value  impacts.  As  a 
result,  the  team  settled  on  a  Windows-compliant 
COTS  EVMS  tool  known  as  winsight  with  which  to 
establish  compatibility.  The  plan  was  to  fit  the  TPM 
tool  later  to  PA  once  it  became  Windows  capable. 
Eor  the  moment,  however,  winsight  would  be  the 
product  used  to  calculate  EVMS  impacts. 

To  prove  out  the  methodology,  the  TPM  Project 
Team  decided  that  a  retrospective  analysis  conducted 
against  actual  outcomes  on  a  mature  contact  would 
be  the  best  approach.  The  exit  criteria  for  a 
successful  test  would  be  (a)  early  warning  of 
technical  perturbations  in  the  program  before  the  15 
percent  mark  in  the  contract,  (b)  early  warning 
significantly  before  indications  from  traditional 
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performance  measures,  and  (c)  calculated  cost 
projections  that  were  statistically  significant  when 
compared  against  actuals. 

At  the  end  of  November  1995,  the  Program 
Executive  Officer  (PEO)  gave  his  approval  to  the  TPM 
Project  Team  to  use  the  T45TS  Cockpit  21  project  as 
its  proof-of-concept  test  bed  and  the  project  began  in 
earnest.  The  Cockpit  21  program  was  managed  by  the 
Jet  Plight  Training  Program  Office  (PMA  273)  under 
PEO(A).  The  program  involved  the  development  and 
installation  of  a  digital  cockpit  into  the  T-45A  aircraft. 
Previously,  this  aircraft  was  procured  using  analog 
cockpit  instrumentation.  The  new  cockpit  has 
improved  training  effectiveness  by  enhancing  the 
cockpit  with  a  digital  data  bus  and  multi-function 
displays  that  will  more  closely  resemble  features  on 
an  increasing  number  of  newer  fleet  aircraft. 

A  three-year  letter  contract  was  awarded  on  29 
May  1992  to  McDonnell-Douglas  (MDA)  as  the  prime 
contractor,  however,  this  contract  was  not  definitized 
until  March  1994.  The  delay  in  defmitization  was 
attributable  solely  to  pricing  issues.  The  system 
requirements  were  very  well  defined  and  had  no 
impact  on  contract  definitization. 

Smith’s  Industries  of  the  United  Kingdom  was 
the  subcontractor  awarded  a  major  portion  of  the 
software  development  work  for  the  digital  cockpit. 
The  contract  between  MDA  and  Smith’s  was  Firm 
Fixed  Price  (FFP).  After  some  time  for  ramp-up,  work 
on  Cockpit  21  development  commenced  in  full  in  July 
1992. 

The  Cockpit-21  program  provided  an  excellent 
analysis  environment  with  several  years  of  Cost 
Performance  Reports  (CPRs)  and  technical 
performance  data  that  included  comprehensive 
planning  and  reported  actuals.  The  program’s 
reported  cost  and  schedule  information  showed  good 
performance  even  in  the  Defense  Acquisition 
Executive  Summary  (DAES)  system,  but  the  products 
being  developed  were  clearly  not  meeting  the 
schedule.  Technical  perturbations  were  being 
experienced  that  were  not  being  reflected  in  the  cost 
reports. 

To  ensure  that  the  TPM  Project  Team  could 
conduct  the  study  in  an  objective  fashion,  certain 
information  was  withheld  by  the  program  office. 
Specifically,  this  included  the  renegotiated  contract 
ceiling  that  had  recently  been  settled,  and  additional 
cost  risk  identified  by  the  prime  in  negotiations.  Also, 
only  one  interview  was  conducted  with  the  program 
technical  representative  to  determine  the  risk  items 
under  the  TPP  selection  criteria.  That  person  was 
asked  specifically  to  express  his  judgment  and 
concerns  as  he  recalled  them  at  contract  award.  This 


condition  would  at  least  bound  the  analysis  to  a 
single-blind  approach.  Otherwise,  only  those  formally 
reported  metrics  were  used  to  load  into  the  software 
tool  developed  to  support  the  methodology. 

The  Display  Generation  WBS  element  was  the 
largest  single  budgeted  item  reported.  Since  this 
directly  related  to  the  software  development  of  the 
Cockpit  21  digital  displays,  known  as  the  Display 
Electronics  Unit  (DEU),  the  Software  Development 
Plan  (SDP)  was  reviewed  for  documentation  on 
software  metrics.  A  good  set  of  metrics  was  reported 
and  these  were  used  in  the  test  set.  The  SEMP  also 
contained  metrics  on  Flight  Test  Problem  Reporting 
and  these  were  also  included.  Other  metrics  were  also 
found  for  Reliability,  Cooling  Requirements,  and 
Electrical  Power.  These  elements,  however,  did  not 
meet  the  criteria  for  selection  of  TPPs  and  were  not 
included  in  the  test.  In  any  event,  since  technical 
achievement  in  these  areas  exceeded  the  technical 
progress  plans,  they  would  not  have  affected  the 
results  of  the  study. 

Test  results  ...  At  the  end  of  January  1996,  the 
test  results  on  the  Cockpit  21  program  were  formally 
reported  to  the  PEO.  The  analysis  showed  that: 

(a)  Technical  perturbations  in  software  development 
containing  significant  cost  risk  are  first  identified 
in  November  1992  before  the  15  percent  point  in 
contract  performance. 

(b)  These  results  are  consistent  from  November 
1992  and  throughout  1993,  providing  an 
eighteen  month  early  warning  before  other 
performance  measurement  techniques  began 
reporting  mitigating  activities. 

(c)  The  methodology  was  discrete  enough  to 
identify  the  specific  activities  contributing  to 
technical  cost  risk. 

(d)  Calculated  estimates-at-complete  using  the 
winsight  tool  were  within  one  standard 
deviation  of  the  negotiated  contract  ceiling  of 
$68  million  and  the  additional  cost  risk  identified 
by  the  MDA— mitigation  instituted  by  the  prime 
on  the  FFP  subcontract  impacted  total  program 
cost.  All  elements  of  the  study’s  exit  criteria 
had  been  met.^* 

As  a  result  of  the  positive  results  of  the  TPM 
study,  an  internal  Peer  Review  of  the  Cockpit  21 
study  results  was  conducted  from  February  to  April 
1996.  Members  for  the  Peer  Review  Committee  were 
assembled  from  the  Naval  Aviation  Systems 
Command  (NAVAIR)  cost  analysis  and  systems 
engineering  competencies,  PEO(A)  program  offices. 
Defense  Systems  Management  College  (DSMC),  the 
Institute  for  Defense  Analyses  (IDA),  and  private 
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industry.  The  TPM  Project  Team  authored  a  white 
paper  and  submitted  their  data  and  results  to  intense 
scrutiny  by  the  group.  On  24  April  1996,  the  peer 
review  group  endorsed  TPM  as  a  promising 
approach,  ratified  the  plausibility  of  the  TPM 
mathematical  concepts,  and  recommended 
prospective  implementation  of  the  methodology  on 
future  contracts 


A  False  Start  and  A  Good  Start. 


As  a  result  of  the  positive  results  from  Cockpit 
21,  and  the  recommendation  for  prospective 
implementation  by  the  peer  review  group,  the  Navy’s 
Stand-Off  Land  Attack  Missile-Expanded  Response 
(SLAM-ER)  program  sought  to  implement  the  TPM 
methodology  to  identify  and  mitigate  existing  cost 
and  schedule  perturbations  and  to  avoid  future  ones. 
The  program,  being  41  percent  complete,  was  thought 
to  provide  a  unique  opportunity  for  applying  the 
methodology  both  retrospectively  and  prospectively. 
In  hindsight,  however,  this  factor  was  the  main  barrier 
to  implementation. 

The  retrospective  application  of  a  process 
improvement  tends  to  be  costly  and  requires  changes 
to  existing  procedures.  The  process  also  becomes 
external  to  the  system  and  excludes  stakeholders  from 
critical  decision-making  in  the  application  of  the 
process— it  documents  history  and  nothing  more.  As 
a  result,  rather  than  enhancing  existing  processes, 
application  of  TPM  on  SLAM-ER  would  have  been 
prescriptive  on  both  the  program  office  and  the  prime 
contractor.  In  addition,  requirements  for  technical 
performance  data  were  eliminated  from  the  contract 
deliverable  items  under  the  aegis  of  “streamlining” 
and  the  program  office  was  not  willing  to  invest 
additional  resources  in  implementing  the  process. 

Lessons  learned  ...  The  lessons  learned  from 
SLAM-ER  were  instmctive  and  led  to  adjustments  in 
the  manner  in  which  the  TPM  methodology  would  be 
applied  in  future.  Only  new  contracts  would  be 
selected  for  implementation.  In  addition,  a  feasibility 
study  would  be  conducted  prior  to  commitment  of 
project  resources  and  costs  for  implementation  would 
be  identified  up-front  to  the  program  office.  The 
project  would  also  use  standard  risk  management 
retum-on-investment  criteria  to  determine  if  an 
implementation  was  feasible,  and  develop  other 
metrics  to  determine  the  methodology’s  utility. 

In  July  1996  the  TPM  Project  Team  accepted  a 
request  from  the  Eederal  Aviation  Administration’s 
(EAA)  Wide  Area  Augmentation  System  (WAAS) 
program  office  to  apply  TPM  as  a  joint  pilot 
implementation.  The  WAAS  program  is  developing 


an  enhanced  Global  Positioning  System  (GPS)  to 
provide  accurate  aircraft  reporting  anywhere  on  or 
near  the  surface  of  the  earth.  The  system  will 
ultimately  satisfy  the  requirements  of  all  phases  of 
flight,  including  precision  approach  and  landings.^^ 

The  prime  contractor  is  Hughes  Information 
Systems  Company  of  Eullerton,  California.  The 
contract  value,  negotiated  in  October  1996,  is 
approximately  $500  million.  The  TPM  database  for 
this  effort  has  been  populated  and  implementation 
continues  with  the  first  reported  cost  reports.  The 
initial  results  are  encouraging. 

Before  contract  award  the  program  office 
committed  itself  to  ensure  that  a  comprehensive  TPM 
approach  that  included  the  PEO(A)  methodology 
would  be  properly  implemented  and  funded.  The 
TPM  planning,  which  included  Hughes  personnel 
through  the  construction  and  verification  of  TPPs, 
progress  plans,  and  probability  distributions  was 
exhaustive  and  required  contractually-defined 
deliverables  (CDRLs).  The  approach,  because  it 
involved  the  substitution  of  electronic  for  more  labor- 
intensive  paper-based  SEMP  deliverables,  has  had 
the  effect  of  reducing  effort  involved  in  CDRL 
delivery.  The  WAAS  program  office  also  included 
TPM-informed  information  in  the  Performance 
Evaluation  Plan  (PEP)  as  a  determining  factor  in  the 
contract  award  fee  arrangement. 

Preliminary  conclusions  ...  Some  preliminary 
conclusions  can  be  drawn  from  the  WAAS 
experience.  Eirst,  the  contractor-government  program 
team  have  gained  insights  into  the  relationship 
between  cost,  schedule,  and  technical  achievement 
issues  that  would  not  have  otherwise  been  available. 
Second,  the  TPM  approach  is  compatible  with  and 
complementary  to  existing  TPM  and  risk  assessment 
methods  used  by  the  prime  contractor.  Third,  the 
initial  results  from  the  first  cost  reports  indicate  that 
technical  perturbations  revealed  by  TPM  reinforce  an 
interactive  environment  supportive  of  the  IPT 
structure.  Eourth,  costs  and  efforts  associated  solely 
with  the  implementation,  while  requiring  significant 
advance  planning,  are  marginal. 


Conclusion. 


The  PEO(A)  TPM  methodology  provides  a 
promising  first  step  in  the  integration  of  cost, 
schedule,  and  technical  performance.  It  achieves  this 
goal  through  a  flexible  and  robust  methodology  that 
is  of  practical  use  to  both  government  and  commercial 
program  managers  and  their  teams.  This 
methodology  has  evolved  over  time  and  will  continue 
to  do  so  as  current  and  future  implementations 
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provide  additional  insight  into  proactive  technical  risk 
management. 

As  an  interactive  and  integrated  diagnostic  tool, 
TPM  promises  to  provide  necessary  insight  into 
those  issues  of  importance  to  IPT  members,  thereby 
supporting  the  concept  of  distributed  empowerment, 
and  to  provide  sufficient  early  warning  of  technical 
cost  risk  to  allow  for  early  mitigation  and  cost 
avoidance.  It  also  promises  to  provide  an  integrated 
tool  to  the  business  of  program  management  that  will 
support  the  new  management  paradigm  of  functional 
integration  and  systems  thinking. 
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